AX Evals Are the Operating System for Agent Rollouts

An agent can score well in a lab and still fail the first week it meets your real repository.

That gap is where Technical Program Managers now live. Model benchmarks tell you what the model did on someone else's task. An Agent Experience eval, or AX eval, tells you what your agent does inside your harness, against your tools, with the permissions and environmental constraints your operators actually use.

Microsoft's
, published July 15, gives this work a useful operating shape. It defines six conditions for an eval that produces signal instead of a convincing spreadsheet. The lesson is practical: the eval is not a scorecard added after the build. It is the program control that tells you what to fix before scope expands.

The number that matters

The number is five. Microsoft recommends running each scenario at least five times because one agent run can be an outlier. A single success is not evidence of a reliable workflow. It is one sample from a nondeterministic system.

For a TPM, this changes the review question. Stop asking, "Did the agent complete the task?" Ask, "How often did it complete the task correctly, under the conditions our operators will face, and what did it do when the happy path broke?" The difference is small in a meeting and enormous in production.

Build the scenario set from incident history rather than an imagined ideal flow. Start with one routine task, one ambiguous task, one permission failure, and one interrupted run that must resume without losing state. Record the expected result before the first run. Then give every scenario an owner who can decide whether a failed criterion blocks scope. Otherwise the test suite will accumulate warnings that nobody is accountable for clearing.

This is not a lab ritual. It is the same move TPMs make with deployment canaries and operational readiness reviews. You turn a broad claim about reliability into a small set of observable conditions, then attach a release decision to each condition.

The framework

I would run an AX gate through four program controls.

1. 
   Use a prompt that a developer, support engineer, or analyst would reasonably type. Do not lace the scenario with evaluation metadata or hidden hints. Microsoft's article calls these representative prompts. If the prompt only works because the test author wrote it, you are measuring the author.
2. 
   A criterion should check whether the agent used the technology correctly and whether the resulting work runs. A file's presence is not enough. For a code change, inspect behavior. For an incident workflow, inspect escalation and rollback. For a customer operation, inspect the final state and the audit trail.
3. 
   User names, workspace paths, operating systems, cached credentials, and prior context can change agent behavior. Microsoft separates a clean environment from a representative one for good reason. You need both. Clean tests reveal hidden leakage. Representative tests reveal whether the workflow survives the mess your users already have.
4. 
   Docker's [AI Engineer World's Fair recap](https://www.docker.com/blog/ai-engineer-worlds-fair-2026-the-runtime-is-where-agent-trust-is-won/) puts the operational pieces in one line: evals, loops, harnesses, context, memory, isolation, and cost. The eval finds the failure. The runtime contains it. If a failed criterion cannot stop a release, reduce permissions, or route to a human, you built a report rather than a control.

This is also why the harness matters. In Lenny Rachitsky's
, Claire Vo walks through a custom Claude Agent SDK harness for Sentry bug triage. She encoded permissions, connected Sentry and Linear, and turned a repeatable task into something an agent can run more consistently. The model is one component. The harness determines what the component can see and do.

What this does not solve

Five successful runs do not predict every production state. The gate reduces uncertainty. It does not erase it. High consequence actions still need bounded permissions, a human escalator, and rollback.

You can remove environmental noise so thoroughly that the scenario stops resembling production. Run the clean case to find contamination, then run the representative case to find friction. Treat disagreement between them as a signal, not an inconvenience.

Teams often reward task completion while ignoring cost, latency, or operational damage. An agent that resolves a ticket by disabling a control may pass a shallow criterion. Review criteria with the same care you apply to acceptance requirements.

The signal that matters most

The strongest signal is that the measurement surface and the runtime surface are converging.

Microsoft is teaching teams how to build AX evals. Docker is framing runtime isolation and harness design as the place trust is won. Cloudflare launched
on July 13 to detect agentic behavior using continuous client side signals. These are different products pointing at the same program problem: agents need observable behavior, controlled execution, and a decision rule for what happens next.

That gives TPMs a concrete job. Create an AX register for every production agent. Name the top workflows, representative environments, pass criteria, repeat count, permission boundary, failure owner, and rollback action. Review it whenever the model, harness, tool surface, or runtime changes.

An AX register should expose coupling between the agent and the rest of the program. If a model upgrade changes tool choice, the harness owner reruns the affected scenarios. If an identity policy changes, the security owner reruns permission failures. If a repository migration changes paths or build commands, the platform owner reruns the representative environment. The register turns "we changed something" into a bounded retest plan instead of a full regression panic.

Do not wait for a benchmark committee to tell you the agent is ready. Run the work your organization depends on, in the environment it uses, and make the result control scope. That is how an eval becomes an operating system for the rollout.

Send one AX scenario your team uses, plus the failure it is meant to catch, to the doronkatz.com TPM desk via LinkedIn. The best patterns will be collected into a practical agent rollout checklist.